Feature - Epoch boundary recompilation phase #33477
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Lichtso
force-pushed
the
feature_transition_recompilation
branch
3 times, most recently
from
c74fd46 to
496b3d8
Compare
Lichtso
commented
Oct 2, 2023
Lichtso
commented
Oct 2, 2023
Lichtso
force-pushed
the
feature_transition_recompilation
branch
4 times, most recently
from
2cd41cf to
b82339f
Compare
pgarg66
reviewed
Oct 2, 2023
Codecov Report
@@ Coverage Diff @@
## master #33477 +/- ##
=========================================
- Coverage 81.9% 81.9% -0.1%
=========================================
Files 809 809
Lines 218062 218105 +43
=========================================
+ Hits 178672 178684 +12
- Misses 39390 39421 +31 |
Lichtso
force-pushed
the
feature_transition_recompilation
branch
3 times, most recently
from
e5d13d3 to
0fac427
Compare
Lichtso
force-pushed
the
feature_transition_recompilation
branch
3 times, most recently
from
7a850e0 to
b9c2b05
Compare
Lichtso
force-pushed
the
feature_transition_recompilation
branch
from
b9c2b05 to
3a9ca95
Compare
Lichtso
force-pushed
the
feature_transition_recompilation
branch
3 times, most recently
from
a29f8cc to
0d25f22
Compare
Lichtso
force-pushed
the
feature_transition_recompilation
branch
from
0d25f22 to
a5814fe
Compare
pgarg66
reviewed
Oct 26, 2023
| let mut loaded_programs_cache = new.loaded_programs_cache.write().unwrap(); | ||
| loaded_programs_cache.replenish(key, recompiled); | ||
| } | ||
| } else if new.epoch() != loaded_programs_cache.latest_root_epoch |
There was a problem hiding this comment.
Could a malicious leader trigger this check by adding a bank in a future epoch? Does it make sense to make this check more specific (e.g. new.epoch == loaded_programs_cache.latest_root_epoch + 1, and slot is within N slots in the new epoch)?
There was a problem hiding this comment.
Aren't leaders bound by the leader schedule in that they can't just submit a block for any slot, but only the one they were assigned to?
There was a problem hiding this comment.
I remember seeing this in one of the outages where the leader submitted a slot way in future. Maybe its not an issue anymore due to other checks.
Would making this check more strict cause any problems? It can be done in a separate PR if that's preferred.
There was a problem hiding this comment.
Would making this check more strict cause any problems?
Probably not, but haven't thought it through.
Lichtso
force-pushed
the
feature_transition_recompilation
branch
from
a5814fe to
5347856
Compare
pgarg66
reviewed
Oct 27, 2023
Lichtso
force-pushed
the
feature_transition_recompilation
branch
2 times, most recently
from
8e7c67d to
1bad18d
Compare
Lichtso
force-pushed
the
feature_transition_recompilation
branch
from
1bad18d to
a5db6d5
Compare
pgarg66
approved these changes
Oct 27, 2023
mergify bot
pushed a commit
that referenced
this pull request
Nov 9, 2023
* Adds LoadedPrograms::upcoming_environments. * Moves LoadedPrograms::prune_feature_set_transition() into LoadedPrograms::prune(). * Adds parameter recompile to Bank::load_program(). * Sets latest_root_slot/epoch and environments in Bank::finish_init(). * Removes FEATURES_AFFECTING_RBPF list. * Adjusts test_feature_activation_loaded_programs_recompilation_phase(). (cherry picked from commit a9509f5) # Conflicts: # runtime/src/bank/tests.rs
Lichtso
added a commit
that referenced
this pull request
Nov 9, 2023
* Adds LoadedPrograms::upcoming_environments. * Moves LoadedPrograms::prune_feature_set_transition() into LoadedPrograms::prune(). * Adds parameter recompile to Bank::load_program(). * Sets latest_root_slot/epoch and environments in Bank::finish_init(). * Removes FEATURES_AFFECTING_RBPF list. * Adjusts test_feature_activation_loaded_programs_recompilation_phase(). (cherry picked from commit a9509f5)
Lichtso
added a commit
that referenced
this pull request
Nov 9, 2023
…) (#34003) Feature - Epoch boundary recompilation phase (#33477) * Adds LoadedPrograms::upcoming_environments. * Moves LoadedPrograms::prune_feature_set_transition() into LoadedPrograms::prune(). * Adds parameter recompile to Bank::load_program(). * Sets latest_root_slot/epoch and environments in Bank::finish_init(). * Removes FEATURES_AFFECTING_RBPF list. * Adjusts test_feature_activation_loaded_programs_recompilation_phase(). (cherry picked from commit a9509f5) Co-authored-by: Alexander Meißner <[email protected]>
anwayde
pushed a commit
to firedancer-io/solana
that referenced
this pull request
Nov 16, 2023
…na-labs#33477) (solana-labs#34003) Feature - Epoch boundary recompilation phase (solana-labs#33477) * Adds LoadedPrograms::upcoming_environments. * Moves LoadedPrograms::prune_feature_set_transition() into LoadedPrograms::prune(). * Adds parameter recompile to Bank::load_program(). * Sets latest_root_slot/epoch and environments in Bank::finish_init(). * Removes FEATURES_AFFECTING_RBPF list. * Adjusts test_feature_activation_loaded_programs_recompilation_phase(). (cherry picked from commit a9509f5) Co-authored-by: Alexander Meißner <[email protected]>
anwayde
pushed a commit
to firedancer-io/solana
that referenced
this pull request
Nov 16, 2023
…na-labs#33477) (solana-labs#34003) Feature - Epoch boundary recompilation phase (solana-labs#33477) * Adds LoadedPrograms::upcoming_environments. * Moves LoadedPrograms::prune_feature_set_transition() into LoadedPrograms::prune(). * Adds parameter recompile to Bank::load_program(). * Sets latest_root_slot/epoch and environments in Bank::finish_init(). * Removes FEATURES_AFFECTING_RBPF list. * Adjusts test_feature_activation_loaded_programs_recompilation_phase(). (cherry picked from commit a9509f5) Co-authored-by: Alexander Meißner <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
This is a different approach to the problem as #32172 was flawed. It ended the recompilation phase at the first block after the epoch boundary, but we need to wait for a rerooting to be sure that we won't replay any blocks from before the epoch boundary anymore.
Summary of Changes
LoadedPrograms::upcoming_environments: Which isSomeduring the recompilation phase, even if there is no change to any environment.LoadedPrograms::prune_feature_set_transition()intoLoadedPrograms::prune(): Delays it to happen on the first rerooting after the epoch boundary.FEATURES_AFFECTING_RBPFlist: Because we can now automatically detect changes by usingBuiltinProgram::eq().Bank::load_program(): Which isSomewhen called from within the recompilation phase.